DOCUMENT RESUME 



ED 442 113 



CS 217 161 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 

IDENTIFIERS 



Werder, Carmen M. 

The Wicked Old Witch Is Dead: Getting Rid of a Flawed 
Writing Exam. 

2000-04-15 

1 6 p . ; Paper presented at the Annual Meeting of the 
Conference on College Composition and Communication (51st, 
Minneapolis, MN, April 12-15, 2000) . 

Reports - Descriptive (141) -- Speeches/Meeting Papers (150) 

MF01/PC01 Plus Postage. 

Higher Education; *Student Evaluation; Student Needs; 
*Writing Evaluation; *Writing Laboratories; *Writing Tests 
Faculty Attitudes; *Rising Junior Examinations; Test 
Revision; *Western Washington University 



ABSTRACT 



This paper reports on the myriad difficulties in getting rid 
of an unsatisfactory writing examination in place at Western Washington 
University. First, the paper provides background information on how and why 
the exam (the Junior Writing Exam) came to be adopted in the 1980s (as a 
diagnostic test only) and explains what the exam was meant to examine. Then, 
it recounts how the examination became a required pass in 1991, with a 
significant failure rate of 40% by 1992. The writing program directors tried 
to improve instructional support services and instructor training since the 
students who failed became demoralized and bitter. The drain on Writing 
Center resources became alarming, and the center's personnel were desperate 
for a solution. This "rising junior" test was finally replaced with a 
different type of exam, one that was not decontextualized and lacking 
validity but, instead, an authentic writing task. It finds that, instead of 
concentrating on fixing or eliminating an exam, educators at Western 
Washington are now launched on establishing an interdisciplinary community of 
inquiry involving faculty, administrators, and students as part of the 
Carnegie Campus Conversations Program. (NKA) 
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THE WICKED OLD WITCH IS DEAD: 

GETTING RID OF A FLAWED WRITING EXAM 

Carmen Werder 

When Richard Haswell e-mailed me after last year’s 4Cs and asked if I was 
interested in joining a panel to discuss mandated writing assessment - to speak from my 
experience and success - 1 was pleased at the prospects of a forum in which to talk about 
our campus’s recent victory over a rising junior exam. Still flushed from the triumph, I 
accepted the invitation. Mainly, I just wanted to understand better exactly what had 
accounted for that victory. I would like to think our success in displacing the flawed test 
was because I single-handedly overpowered the creature. Though God knows I tried to 
sneak up on it in the night and smother it, I know that its death resulted from the work of 
many people who exerted a lot of hard work and endured. But hard work and endurance 
don’t necessarily spell success. What strategies did we use that worked? I have spent the 
last two years trying to figure that out. So I come to you, not with some master plan for 
success, but with an exploratory analysis of the case of our test’s demise - The Case of 
the Wicked Old Witch from Western Washington University - in the hopes that it will be 
suggestive of how we composition folks can, in fact, take back the test. Though the Oz 
metaphor here is meant to be playful, and does highlight some serious recommendations I 
will mention later, I do not intend for it to trivialize this topic. Getting rid of our writing 
test is the most difficult thing I have ever attempted in academe. 

Let me begin by describing the context which prompted the creation of our test 
because it came into existence initially - not by legislative mandate - but prompted by an 
offer from the corporate sector. In the early 80’s representatives from a corporation that 
hired many of our graduates approached the administration offering to pay for remedial 
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writing help for them. It turns out that while they found our graduates attractive 
employees in many ways, they were concerned about their writing skills. Well, you can 
imagine the administrative scurrying that occurred in response. A campus writing 
committee was appointed to address the problem, and they decided to implement an 
upper-division writing proficiency requirement. At the time Western students had only 
one required writing course, English 101, as part of their general education requirements. 
So the plan was to require a second writing course at the upper level, preferably in the 
major, which would serve as a graduation requirement. However, the Committee decided 
there needed to be some kind of screening device to determine whether students were 
ready to benefit from the instruction to be provided by this WP course. In part, they were 
concerned about the large number of students who transfer into Western. While there 
was no actual data that transfer students had inferior writing skills, there was a 
widespread assumption that they were not as proficient as our native students. The 
Committee’s general sentiment was that there needed to be some kind of instrument at 
the junior level to catch transfers as well as to catch any native students who were not 
ready to proceed to the WP course. 

They recommended implementation of a “rising junior exam” so called because it 
was intended for students to pass as they moved from sophomore to junior status. And it 
was also a barrier writing exam because students had to take it before enrolling in a WP 
course. Together, the test and the course would constitute the writing proficiency 
requirement for graduation. In 1983, they instituted the Junior Writing Exam as an 
informational, diagnostic exercise only; in other words, students were required to take it, 
but not necessarily to pass it. The idea was that those who received an “Unsatisfactory” 
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would seek out remedial help in the Writing Center or in a 200-level writing course that 
had been created with the test in mind. So the Junior Writing Exam (the JWE) didn’t 
start out to be a mean thing, but rather a means for students to obtain information about 
their writing ability before moving on to a writing proficiency course. 

The JWE was a two-part test. The first portion was a 45-item multiple-choice 
identification of sentence level errors; it required just over 50% ( a score of 23) to pass, 
and it was graded by a scantron. The second part was a writing sample, a one-paragraph 
summary of a 500-600 word passage drawn from readings on various topics; it was rated 
either Satisfactory or Unsatisfactory based on a set of primary traits generated by the 
writing committee, and it was evaluated by a cadre of faculty graders from across the 
disciplines. 

The JWE received negligible attention until the fall of 1991 when it became a 
required pass . Up to this point, since the test had been merely advisory, students 
generally did not take it seriously. While some students undoubtedly did look at their test 
scores and sought out more instruction, many simply went through the motions as 
evidenced by a growing number of summaries which were not paragraphs, but more in 
the vein of editorial comments on the test itself - test graffiti actually. One could argue 
that such students were rhetorically astute considering the summary’s contrived rhetorical 
purpose. Truth is, it didn’t really matter what they wrote on the summary - just that they 
took the exam. Those who did take the summary seriously and failed it could seek 
additional information by making an appointment with the Assistant Director of the 
Writing Center to review their exam, but there was little incentive for students to do so. 
Whether students failed or passed, they simply received overall scores, no specific 
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information on how to improve. In the late 1980’s, with increasing pressure from our 
state legislature to focus more on assessment, our Higher Education Coordinating Board 
formalized this concern with a mandate to all state baccalaureate schools which stipulated 
intermediate writing assessment. 

When the JWE became a required pass in 1991, it became what we now know as 
“high stakes” writing assessment because students needed to pass it not only before 
enrolling in a writing proficiency course, but in order to graduate. Soon the test assumed 
a life of its own with a growing appetite for consuming more resources. 

In the fall of 1992, a neophyte in writing assessment and a doctoral candidate in 
rhetoric and comp, I assumed the part-time position as Assistant Director of the Writing 
Center, with my sole responsibility to respond to the students who failed the test and 
needed remedial help. Dorothy plunked down in Kansas all right, but without the benefit 
of Toto and friends. Naively thinking that this position was about writing instruction, I 
began walking what was to become a long and windy road. 

By the time I entered the scene, the failure rate on the test overall had become 
significant - averaging about 40%, with the failure rate on the summary portion hitting 
an all time high at 54% in the previous quarter. The administration had increased the 
Assistant Director position to % time in an effort to respond to the large number of 
students who had failed the test. The high fail rate was compounded by the fact that 
students could take the exam only once each quarter, so if they failed one or both parts, 
they could not re-take the test until the next quarter. The JWE was quickly becoming a 
bottleneck. 
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After reviewing the disturbingly high failure rate and attending my first scoring 
session, I began a campaign to improve the test itself, focusing first on improving the 
training of graders. Instead of viewing the summary simply as an indicator of whether or 
not writers could profit from more writing instruction, graders tended to evaluate the 
summaries as exit tests. For example, if the summary had evidence of imprecise word 
choice or prejudicial misspellings, graders tended to award it a U, never mind the 
established criteria. I watched a “horn” effect - a perverse halo - with graders punishing 
stylistic infelicities with failing marks. Citing the need for high expectations, many 
graders vented all their frustrations about weak writing proficiency - not just about a 
particular student’s summary - but all over the world. 

Concerned, I volunteered to help conduct future grading sessions and made some 
changes. An evaluation rubric was stamped on each summary sheet for graders to check 
off. Before scoring, graders reviewed a packet of sample summaries which had already 
been scored and glossed. There was little or no room for faculty discussion. The stakes 
had become too high. These changes in the training sessions did result in increasing the 
pass rate, but there was much grumbling that we had lowered our standards. 

At the same time, I tried to improve the instructional support services because the 
test results had such a demoralizing effect on students who failed. I began by asking 
graders to use “S-U” scoring on the tests themselves because it was much easier for 
students who failed the summary and came in for consultation to listen my 
recommendations if they saw a U and the scoring grid on their test sheets than to be 
greeted with a big “F.” I created a summary writing workshop that was offered several 
times a week through the Writing Center, and videotaped for our off-campus sites, and 
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which further helped increase the pass rate. The Writing Center program manager and I 
beefed up the JWE study packets providing sample tests, answers and model summaries, 
test-taking strategies, and additional resources. We even rewrote test summary passages 
so that they included explicit thesis statements. We stocked the Center with sample 
passages for practice and trained the writing assistants to teach to both parts of the test. 
The Writing Center became the JWE Center. And for awhile I thought we were making 
headway: by 1994 the pass rate had stabilized at 75-80%. But student resentment against 
the test continued to run high. And I was so busy trying to keep up with individual test 
consultations and the myriad details involved with sustaining the exam, that there was 
little or no time to reflect or consider the test’s implications. Eventually, though, I began 
to look beyond the test. 

Not only had the Writing Center become aligned with all the negative response 
from students who had failed the exam, but also the limited nature of the exam had 
served to convey a reductive sense of writing with serious curricular implications. 
Writing became equated with the JWE which became equated with finding errors and 
writing one-paragraph summaries. While this notion of “writing” bore little relationship 
to the writing as critical inquiry model that we were trying to inculcate into our general 
education writing courses, the isolated summary write became common in the first year 
writing course and beyond. About this time, another logistic nightmare occurred: The 
writing committee decided to implement a computer block on the registration of students 
who had not passed the test by the time they had accrued 120 credits - a credit block that 
could only be removed by coming to the Writing Center and completing a contract form 
committing to certain activities before the block could be lifted. Angry because they 
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could not register for any classes (not simply a WP course), students became even more 
hostile toward the exam. 

While I was beside myself by now because of the drain on Writing Center 
resources, I was equally alarmed by the harm the test was doing to students’ attitudes 
about writing. Eventually, the moment of truth came when a young Hispanic woman, a 
senior, who had failed the multiple-choice part of the exam four times came to my office 
to re-take that part of the test. Since she had such high test anxiety, I got permission to 
administer the test individually in an effort to calm her down. It was the first week of 
June and she showed up with her graduation paraphernalia in anticipation of the 
ceremony the following weekend. Now I was anxious. What if she failed the test again? 
After she finished, I counted up her correct answers and much to our mutual relief, she 
scored the requisite 23 in order to pass. I confess I don’t know what I would have done 
had she failed it, especially knowing how little the 23 meant. I do know that as she left 
my office jubilant in what I saw as a hollow victory, I had to question how we had helped 
this student learn. What perverse sense of writing gains was she leaving with? And how 
could this test be justified? I vowed to shift attention away from the test itself to the 
institutional structure that was keeping it alive. 

I turned first to the campus writing committee. While I had regularly attended and 
worked with this group, I was not a member of the committee. And my increasingly 
vocal critique of the test had begun to incur the wrath of a supervisor who had helped 
institute it and who was an ex-officio member of the committee. While I acknowledged 
that the test had reflected the best assessment thinking available when instituted, I 
suggested how it was not serving students or faculty or writing or writing instruction, but 
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only its insatiable hunger for more resources. With enormous relief, I read about the 
problems with rising junior exams in Ed White’s Teaching and Assessing Writing- sure 
that he was describing our test - and tried to insinuate his concerns into the committee. 
But to no avail. Finally, when Brian Huot’s CCCs piece ‘Toward a New Theory of 
Writing Assessment” was published in Dec. 1996, 1 promptly copied it and sent it around 
to the whole committee - sure that his cogent argument on the need to link an assessment 
instrument with writing pedagogy would certainly persuade them. Again, they seemed 
unable to conceive of life without the JWE and seemed confident that it was better than 
nothing. My blatant criticism of the test was causing friction and, soon, I received a curt 
memo from my supervisor advising me not to attend the writing committee meetings 
anymore. Up to now, I hope you have heard a cautionary tale. Now to the part about 
success. 

Making Friends with the Tin Man, the Scare Crow, the Lion, and all the Munchkins 

By now, I was desperate. Eliminating a test that has assumed a life of its own can 
require enormous energy and can benefit from combining resources (brains, heart, and 
courage). In my expanded full-time position as Assoc. Director of the Writing Center, I 
replaced my JWE consultation duties with a focus on writing in the general education 
program and began consulting with the gen ed committee. Together, we proposed a 
three-part writing initiative: to institute a second writing course at the lower level, as part 
of our general education requirements; to endorse the writing links (the pairing of a 200- 
level writing course with courses in the disciplines), and to establish a task force to 
explore the creation of a University Writing Program. 



9 



This task force included members from the English Dept., the Writing Center, as 
well as faculty from other department interested in writing instruction. This time, I was 
an official member. In addition to bringing a writing program consultant to campus and 
securing her recommendations for a writing program, the Task Force also sponsored 
cross campus visits to inform all departments about the writing proposals and to seek 
feedback. We also held area focus groups to discuss the findings. In the course of those 
departmental and area conversations, concerns about the JWE emerged. In fact, a faculty 
member from the College of Business suggested replacing the exam with a writing points 
requirement, whereby courses throughout the curriculum that provided writing instruction 
would carry not only course credit, but also writing points, so students could be required 
to earn so many writing points as part of their coursework. In this way, assessment 
would occur in the context of courses where it belonged. Eventually, it was this writing 
points idea that seeded the motion that successfully replaced the JWE. 

Drawing the Curtain on the Wizard 

Just as the big scary voice of the Great Oz came from a little scrawny man, so 
frequently does the mandate behind an exam reveal more rumor than actuality. Find out 
what the mandate actually says can be quite revealing. By the time the cross- 
departmental meetings were happening, I had become a member of the Assessment 
Advisory Committee. When one of its members referred to the oft repeated mantra -We 
really have no choice but to have some kind of intermediate writing test because our state 
legislature has mandated it - 1 decided to check out the actual language of the mandate 
with someone I had met from the Higher Education Coordinating Board. Low and 
behold the mandate that everyone had been citing these many years was actually very 
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generic. It said nothing about schools needing to test all students individually at the 
intermediate level. Only that there needed to be intermediate writing assessment. 

Getting Out of Kansas and Oz 

When we are feeling helpless, we tend to keep going back to the same old place. 
Physically and intellectually going to another site can create a new perspective and 
provide new options. I turned to .my neighbors to the east - Washington State University. 
I don’t think that Rich Haswell and Susan Wyche who were responsible for designing the 
portfolio assessment at WSU when they were there even know that they are a part of my 
success story. But they were. After attending an assessment colloquium at WSU, I 
reported back to the Assessment Committee. While they were skeptical about taking on a 
portfolio system, they began to see how the JWE didn’t have to be the only way to do 
intermediate writing assessment. Together, we drafted a motion to eliminate the test, 
which the Assessment Committee passed and forwarded to the writing committee as an 
advisory recommendation. While the EWC ended up being split on the motion, 
committee support for the test was clearly waning. And the Assessment Committee 
dialogue had also gained the attention of the Director of our Office of Assessment and 
Institutional Testing who had been skeptical about dismantling the test, but was 
beginning to see how we might interpret the legislative mandate differently if we chose 
to. 

As a result of his inclusion in the conversation, the Testing Center ran some data 
about the exam which revealed some useful pieces of information: 1) Transfers 
performed at generally the same rate as native students and scored even a bit higher on 
the objective part and 2) the level of difficulty for the objective part was ridiculously low. 
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And before a psychometric expert from the Psych Dept, could convince the Assessment 
Director that the solution was simply to write a “harder” test, I brought in information to 
show how the JWE was flawed because it was not an authentic writing task in that it was 
totally decontextualized and therefore lacked validity. This time, the information from 
Huot’s article about a new model of writing assessment was heard. 

Following the Yellow Brick Road 

Successfully dismantling a flawed exam may well result from indirection. The 
road may lead away from a focus on the writing assessment instrument per se toward 
larger initiatives. The connection with the Director of Testing led us beyond our campus 
committees when in the spring of 1998 we were invited to participate in a statewide 
writing project involving all six public, four-year schools as part of a pilot project to see 
if scoring writing samples from seniors might serve as an accountability measure. In 
response to our State Legislatures’ desire to find ways to measure institutional 
effectiveness, the assessment coordinators had suggested the plan and invited 
composition faculty from each school to respond. While we were resistant at first, we 
participated the first year and were pleasantly surprised to see how useful the discussions 
were for getting at what we value most in learning outcomes. We are still piloting the 
project and are not sure if it will ever serve as an accountability measure, but has 
generated a discipline-base rubric and suggested ways to enhance faculty development 
efforts. So far, we have been able to use what began as an accountability proposal to 
further assessment efforts at our own schools. 

In the context of this expanding conversation both on and off campus, in spring 
1998, a sub-group of the writing committee approached me to help craft a new motion to 
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eliminate the test. After a number of meetings and multiple revisions, this sub-committee 
proposed that the test be replaced with more writing instruction, specifically in the form 
of the writing points plan that had emerged in our focus groups. I must say that it was 
with a great deal of pleasure that I accepted the writing committee’s invitation to speak to 
the proposal when it went before our Academic Coordinating Commission for review in 
the spring of 1998. It passed unanimously and was ratified by the Faculty Senate two 
weeks later, again unanimously. The JWE was administered for the last time two years 
ago this month. 

While the writing points plan is still under study and the second general education 
writing course has been delayed pending additional resources, the elimination of the JWE 
has afforded us the space to address writing assessment together in a broad base way. As 
a result of the expanding conversations about assessment in general, we now have a solid 
alliance between the Assessment Office and our new Center for Instructional Innovation 
(CD). Now in a reinvented position as Director of Interdisciplinary Curriculum and 
Assessment, I am working with the CII to create campus resource teams that will work 
with departments in designing learning outcomes, with special emphasis on our FlGs 
(Freshmen Interest Groups). On our campus, writing assessment has become a much 
more situated activity - as part of a larger enterprise - and not simply as a one-size-fits- 
all writing test. I suspect that the writing program at the University of Missouri at 
Columbia has been so successful because Marty Townsend and her colleagues have 
understood the importance of this kind of situating. 

As part of our campus’s extended conversation, we have discovered an especially 
influential national ally in the Carnegie Scholarship of Teaching community. As Jean 
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MacGregor urges, we need “to think of assessment as a process rather than a particular 
technique or an instrument” and calls for creating “communities of inquiry where 
assessment is part of a larger discussion of learning.” Instead of concentrating on fixing 
or eliminating an exam, we are now launched on establishing an interdisciplinary 
community of inquiry involving faculty, administrators, and students as part of the 
Carnegie Campus Conversations Program. 

Killing the Wicked Witch 

It is tempting to think we can make a bad assessment instrument better. For a 
long time, we tried to fix the test. But if a test is wickedly flawed, such as completely 
decontextualizing the writing task, fiddling with surface qualities will not fix the 
problem. While not as simple as clicking our heels together, we have managed to 
eliminate the test and institutionalize assessment in a better way. As Inga Baird Hill 
emphasizes in a recent issue of Assessment Update, ‘To achieve true success in the long 
Run, universities must become learning organizations.. . Universities themselves must 
become learners.” While we have miles to go before we sleep, I think our university is 
beginning to learn and to commit to memory that assessment must be at the center of that 
enterprise. 
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